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ABSTRACT 



As part, of the cnrrent txend toward adc^ting modem, 
objectiver-staiidardized test methods in evaluating the teaching 
o£ English as a foreign language (EFL) in the schools of the 
Socialist Republic of Romania,, this review of pedagogical lit- . 
erature makes available to the English-reading audience in- 
formation that hais appeared in Romanian materials. It describes 
and evaluates the traditicmal testing system used in Romania, 
with special attention to EFL testing at all educational levels, 
as reported in major journals and a few unpublished documents. 
* The review aims to detail the background against %rhich its 
author was invited by Romanian educational officials to run 
a testing experiment using an American-made objective-standarized 
EFL test upon a population of 201 Romanian students of English. 
Subjects represented three levels: 12th graders, university 
freshmen, and university juniors. The review describes 
the conditions of the experiment as well as the basic statistical 
information the tests yielded (e.g., Pearson r, interpretative 
recommendations based on published norms, ate). Finally # the 
review suggests general guidelines for a grounded theory of 
objective testing of EFL in Romeutia. 
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BOMftNIftN TESTING PRINCIPLES AND METHODS:. A REVIEW OF A^^lII^ 
^^^LITERATDRE CONCERNING THE USE OF OBJECTIVE TESTS OF 

ENGLISH AS A FOREIGN lANGDAGE (EFL) 



In the educational system of the Socialist Reptiblic of Ro- 
mania a pupil- takes periodic oral and written tests as a part 
of lie usual structure of lessons in most subject areas. As he 
passes from one major educational stage to the next^^the^^pupil - 
undergoes his principal examinations in selected subjects. 
<See Figure 1, a schematic design of the current Romanian school 
system.) The nature of these various tests, as described in 
all available literature, will 'be the subject of the fi.st section 
of this report. Subsequent sections will outline current trends 
in testing English as a foreign language at all educational levels 
The final part will briefly describe the refults of the admini- 
stration of an American-made standardized ol^jective test of EFL 
to 201 Romanian students of English in the spring of 1975; 

General Attitudes Toward Education al Testing 

Both the theory and practice of educational measurement 
today in Romania follow the traditional European model in which ' 
-student achievement is evaluated on the basis of classroom per- 
formance, on results of periodic oral and written examinations, 
and on general behavior. The latter is particularly stressed in 
the communist world where conformity,, discipline, and obedience 
are highly valued." (Braham 1972, p. 61) 

in recent years, Romanian testing specialists and some teach- 



ers have carried on a debate concerning types of^ao^nations 
and have even experimented on a limited scale with -objective" 
tests in hopes of improving the validity and reliability of 

ucational measures. (Cerchez and Cerchez 1976; Noveanu 1976; 
Bejenaru 1975; Zetu 1975) Meanwhile, most practicing classroom 
teachers continue to determine student progress on the basis 
of rather impressionistic measures, both in the daily evaluation 
of individual oral recitations in class, hcanework assignments, 
occasional extemporaneous quizzes, and periodic formal written 
examinations, as well as ojLthe .occasion of a student; s_attempt__ 
at the maturity or baccalaureate examination, required of those 
who wish to pursue higher education. (Braham 1963, p. 63) 

The major academic examinations that face students include 
the baccalaureate examination at the completion' of the elementary 
secondary cycle of studies, the competitive admissions examina- 
tion into higher education, for those who do not enter the labor 
force, and the diploma examination upon the completion of the 
last year of post-secondary education. (Braham 1972, pp. 61-62, 
95-96) Since these milestone examinations follow the pattern 
of evaluation described in the previous paragraph as the typical 
method of classroom testing, they tend to reinforce traditional- 
tee i;ing methods because lower-grade teaches wish to prepare their 
pupils for future evaluations as much in examination protocol as 
in material. (See, for instance, Ministerul Invatamintului 1971, 
p. 85, for an outline of the testing formula used in the uni- 
versity admissions examination for summer 1971.) University in- 



solictors, too, tena to xise a coiabinatioa of ^ron5>tu oral . 
recita4pns or translations and periodic ^itl:^ papers (usually 
on literary or interpretative topics for th^se ^ajoring in 
English) as the basis of the marks they give students. 

. . indeed, the established system of testing ^rtualiytevery 
subject by a teacher-devised combihation of oral dnd>_written 
questions is so ingrained that university panels whose job is - 
to screen candidates for admission into spe^fic fields imitate 
the examination style of the elementary and secbndai^ schools 
vhile sometimes complaining that s)fch non-objective, \nonrstan- 
'dardized procedures cannot provide reliable eyidepce about which 
of the multitude of candidates are best qualified vtb ffll the 
govemmentally limited " number of places in each university 
faculty. (Zetu 1975; Bancila and Chitpran 1972) Although most 
persons concerned with educational evaluation agree that the 
development of objective tests would be useful at all levels 
and for all disciplines (with the exception, perhaps, of art), 
the grip of tradition is strong. Even those who advocate stan- 
dardized and objective measures of achievement for classroom 
evaluation of pupils and for university admissions procedures 
often include very subjective measures under the rubric of. 
-objective testing.- (See Section VI, below.) 

jl. Testing Trends in Englis h as a Foreign Lahguase 

^ Testing principles and practices in English as a foreign 
language to a large extent reflect these 'mainstream Romanian 



attit\rfes toward objective t^^W- * 

•„ a recent teacher's guide for use in the 

-^:::rat r^eneraX schooi Xevex U,es e-X.^ 
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^ a grammar, vocabulary or ^ ......sed in 

- ---"°^::r rgrlltive cogent .erhaps in ^ 

tr:tll. well, .ejenaru, ^ 

Romanian) , ne *=> 



approach to classroom testing is understandable since* teachers 
tend to perpetuate the style of tes-tiiig which they experienced 
as students from the elementciry to' the university levels. 
Furthermore, the training teachers receive in pedagogy emphasizes 
an eiaborateu pattern of jlesson plans but is very sketchy about 
"the various types of ^ testing procedures that might be suitable 
for the lessons. It seems taken for granted that teachers will 
intuitively know how to ^^st cuid^'that only general guidelines - 
need be mentioned. A/widely used" pedagogical handbook putlines ; 
the steps of testing 'as follbv/s:. (Ij) annoimce the topic a:nd scope 
of the test, (2) admilnister the te^t, (3) evaluate the results 



of the test, and (4) igive the next hpiheworJk,,ajss±gnment.. (Salade- 

2uid Munteanu, 1971) ^^The applicatipn of these steps is not 
developed at length. '\ 

III Two Impressionistic Experiments in. Testing EFL at the 
Elementary School Level 

Two experiments that involved EFL bi:|t did not use reliable 
instruments for evaluating their results are documented in the 



literature. . 

; ■ - • y -• - 

Oprica^s (1975) experiment with English teaching methods in 
the first grade reports no measure other than a general impression 



that (a) kindergarten pupils who ^^ave studied any foreign language 
are more receptive to foreign language instruction in first grade 
than are those who did not; (b) the use of direct-method/situation 
context training is more suitable for pupils ages 6-7 than are 
other instructional methods; and (c) , as a corollary of (a), if 
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foreign la^g^ge training begins early and continues uninterrup- 
ted -thr;;;Sh all subse^uen^sch^^^^ years, the necessaSy conditions 
to produce functionaT^ililguals will obtain. ; 

Although oprica's project was done on a massive scale^--5.00 
first grade pupils in two schoo'ls-the experiment as reported 
se^s repiete with methodological faults: Each teacher wis dir- 
ected to use whatever techniques seemed to him best suited to 
the occasion. Oprkca reports thJt by rough classification all , 
the techniques employed were either t>,ose of the direct method 
• or audio-lingu^ approach. Classe's were to be taught about^^ 
total of 200 English words related to .concepts they already 
knew in their mother tongue' - e . g . time,; location, colors, etc. 
■ Mihough -not Specified, the gauge used to measure the hypothesxs 
that bacKground In W foreign language makes further foreign 
language learning easier seems to have been the percentage of 
English terms assimilated by pupils with the background as 
compared to the percentage used by pupils without such prior . 

training. Both groups experienced similar treatment in an attempt 
to get at psycholinguistic underpinnings to child acquisition of 
a second language. 

in an e,ilier experimenrto evaluate the effect of using 
the oral/aural method of teaching B;pglish to elementary pupils 
in th» first'through fourth grades, Dumitrescu and Galateanu 

■ (1970) are not at all clear about how they tested the effective- 

■ „e»s of the new treatment. Both agree that their separate exper.- 
«nts indicate the suitability of using audio-lingual tech- 

'1 1 y 



niques in place of traditional graimar-translation ones^ for , 
yotag learners of English, but Dumitrescu discusses only _the 
pattern of drills she employed while Galateanu is xnore-po^ted 
about testihg but only slightly more informative abou€ the "Con- 
ditions of evaluation. 

' At the start of the experiment, Galateanu established tvfb 
piipil groups-one taught by the experimental oral/aural method, 
tL other by traditional techniques-and at the end of the ,year 
she administered identical tests .to the- two groups for compari- 
soit.__.The:first^test_required.pu^^^^^^^^ 
given a coinmand in English (to/test listening, comprehension^ ; . 
in order to test speaking the second required pupils to orally 
describe some, pictures-the pictures are not specified in the , 
report; the third and fourth tests required pupils to read aloud 
ahl then to take down di c tat ibn from a familiar text; in the 
final test pupils had to translate a passage from Romanian into 
English and another passage from English ^^n to Romanian. 

t : • . X >. ^ : 

Galateanu does not explain hov^ she evaluated each perform- 
ance; but she concludes that the experiment*! group did better 
on the command, description and reading tests and the control 
group excelled on tak^g dictation and doing translation. ~ 
Galateanu seems surprised and pleased by the results, without 
-realizing that the two groups simply did better on those tasks 
which they had studied. . 
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nr.. «Mi »Uim agPtrUiwt in Otoi»ctiv Toting ef »rt on th* 



ami mk wrnxxi* Cmf)« fc^ x»«at 4««m»tr«t#« both hmt 
IMidmt«*aiiii9 of objoetlvn t«st« and her «iMll«Tlyio9 concern for 
tMtiAf 9Qm mmtxmxn wtxmmc^tA %a Inikraail* ftiaUn. 
mK99tUmm%, tut ooiy wfwrtod um of «a objccUvo 

«Mt of irt Itt MOMmi*. «»Miplifi«s tlM lov 4«9rM to wfeicH Vailtf 
rollAblo mmtimsxtmrn luiv« /fm *«^tod to MoMolwa elm»mtotm 



■f 

Ouriiiv tiM fifst and soooimI trlMiotsrs of tho lf71*72 «c«- 
tauto tllv* «dininictm<l four briof^aaitiplo-oltoieo stylo 
tooto to yooiif M0oet» i#lio l»oloof«d to two ttli «rodo oloMtary 
a»ilii^ cio«»«« tHo Iftrnm for ooontrwUoo workoro in 
twrturint foot tfatoo woro in Oetobor and Hovoniior If 71 and 
ff«te«nry aaA llarch I3tl, At ono point in hwr retort tiWa aaya 



tiitt* CO ai^jaota, yac later aHa raporta data on only III— 
oaa einsa of 2f» tit* othor of 11. ^ 

•Uvn oat ftar axporinaotal 9oala in tha Corn of fivo qiMttionai 
11} ttiot axaetly alxmld tm |aatod7 
121 ^wpald it Iw tastadT 

1)1 mm can tlia^/objactim of tDat aeiiool fYllalNia b* 
^'^'■a ititli an objact^iva t«at7 * 
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141 Soif Miflit thm irapU* hm |Miycl»logic«lly pr«p«r«d 

f or tJMi tmmtl . . ^ 

fS) iknr will tim tmmt b* ■trttctttr«47 

m «B Muilysis of ttm typM of orxor* oach p«s|ili had Mdo 
,M m • tzmAitiotmX «irltt«ii trawilAtioa pr«to«t ana on • mrim* of 
iftcll«r or«l r^cltUtioiMi, SlXw c«toforl»oa tho mi9t»kmm ttsi»9 
■snrls* atit. p. U) chart ,of JLangu^ «kill» mnA ooapoiMots. 

•rror typao wmtm thm mA thm baJi» of constructing 
fo«r t«fi«itMi Matlpio-ohoioo tosts. 

^ tmlf twt in tbo aoriw that iilirn 4Morib«» con- 
■tr«otod m tlM folloiflng ■•nnori ' *fti« x^imtim ••tarinl In 
tMO »ritisli t«ait» Otemtoy IW anA Al««an4«r 1979} and in tlia 
ttm eliaptara of tlia iMipila* tvxtliook wHich in har opinion traatad ^ 
•iweifie problaiw that ooowrrad In tba prataat. fllvr daviaad a 
\ taii-ita« ■wiUpla-choioa taat ooirarlnt varb fo««* «»odallty» 
imct. Intarrofativa-nafatlva. and paat tanaa) and vocabulary 
tyiMa faMrala, naa»a of daya, and pr^poaitioaal tmita) . Taking 
into oooaidaratloo all aba bad analyaad and prallAinary pi^il 
tmmeUmt to purticlpatlnf in a no«-traJutional taatlnf aapaftbant— 
tbay ^mtm antbaaiaatlc. aha raporta— lilira allotad 10« points to 
ttm taatt 11 points aac* to tba i ifarb atractar* iumm and $ to 7 
polata to tha 4 voeabolary <|aastiona. 

Mora adalnlatarUif this taat to tba mdbimetm, «llva spant 
oooiidarabl* tina faadllarlsla? tba* vitb tba fomt of a iwaltlpla- 
eiM»ioa tast. Sba says that sha aodalad %iL oe«^ styla test ({laastlon 

II 



by wlttng th« following lt«i on th« board: (The •correct- answer 
is under eecrttd.) 

Wittn the teacher g^"j »P 

eoMS In the pupil* 2. SUT down 

cw « fc^K J tx>n*t cere 

The pttpiU were given ten lainutea to take each teat in the seriea. 
Oorlng the rewiind^r of the hour, the pupils were a.k«d to check 
their answer* and 'typical gramatical error.- were eiq»lained on 
the board with what Silv« call, -different structural exercises. " 

The principal conclusions which Silva draws froai the experi- 

awnt include the following: 

(•) The tests are 8iaxi»ally economical in the ti«e it 

takes to check knowledge and to determine "objective 
grades for a large number of pupils j" 

(b) The tests stimilate n-sak st**dnnts to be proinpt and 
to persevere in their studiesi 

(c) The tests are accepted by the pupils as a normal 
pert of the process of the xesscmi 

(d) Errors diminish in successive administrations^ of 
the tests; 

(e) The tests stimulate independent pupil activity and 
creativity in class work: 

(fl The tests are new, ',i£icient and logical, and the;- 
increase the efficient assi-mjlation of a foreign 
language ; 

(g) The tests do not lequire special efforts and. fror 
• technical poin*^^ of view, are ea»y tc construct. 

✓ 15 
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B. Ob— rwfloM on the T eat D^glqn 

•Uv«*s honost «ttM«»t to dowlop « •«rie» o£ obj«ctlv« 
tMUi'of «FL i. -rlott«ly f U*#o4 by « b«.lc «i* jndor.tanding or 
iMk of ««far«n«M of contwit validity, to«t toiiability and pro- 
tmm^ and ccntrol group critaria. Tha faot that laval. of diffi- 
ottlty, di«crl«inatioii and point vaXoa for aach ita« *iara datar- 
wdmA baforc aach taat ma adiOniatarad (on tha baaia of arrora 
p^ila Mda on quita divargant tranaUtion and racitaUon prataata ) 
a ggaata tha dagraa of conf oaion aboot ita. analyai* built into 
«ha taat. 

Itoadlaaa to aay, baaidaa baiag iK>ra a tatt of coi^>ort«ant 
in NoMnian aociaty than a taat of tha Ingliah languagif, t!«i 
ita« oaad to faadiiariaa pupil* %fith thr .ttltlpla-cholco typo 
qwiation la a poor ona alnca aach chcica can ba dafandad on 
l^tacUc and aamntic grounda. «av«^la.a, Silva arguaa that 
• taat cooatructad of auch Itai-, a«mg othar thlnga, -paxmita 
tHa »oat objacuva grading poasibla.- Cp. 70> rurthar«ora, m^nf 
of tha tan itaa«i uaad on tha irarb/irocabulary taat alao contain 
aiallar conatruction problaw. ror inatanca, tha folloifi»g itmm 
axo typical . It«i fozaat ia Silva* at / 

You flniahad your laasonat now i* 25 

you go to tha cinaM. *• gJL 
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jathough in her conclusion. Sllv. cI^Imb that the re.ulf 
of Uter te.t« mho^ .tudent. to be getting better In English, 
given these types of questions (so arbitrary or obvious In their 
«i«ier.), one wonder, how valid or reliable Sllva's findings are. 
The fact that so few Items are used to cover such a broad range 
of gra«»atlcal and lexical concerns Is Itself enough't/* raise 
Mirious quftstdon* 

int. rMtln,ly, «"« 1. ""'ful in her r««rt in figuring 
th. p.rc«.t.,. of .t»d«.t. in «eh cl... «ho ia.«d «cl. t-.t 
ium. (Sh. do., not con.ia.r ho- -any ..l«t«l ..ch cholc 

IB on. in.t«c. .h. ignor.. th« fct th.t on. <l»..tion 
U by 7»% in oa. el— «d by 10«» i« th. oth« cl... «a 

rt. loo. to9.th«- th. «»«. on thi. vty difiicult it- with 
tb. Mch b.tt.r «or.. on th. n«t t« it— (.11 thr« of which 
*«l with -od.1 ».rb.). .no th.n conclnd.. th.t -th. .tmetur.l 
grWMticl .l—nt. of ui mc6Ml for- tq«..tion. 1. i. ») b.« 
b-n «.l-il.t.d by .M.roKi-.t.ly half of th. tot.1 |»?il..- 
(p. 71) ThTM lt».~wn W.11 oon.tr«!t»« o«»~c«. h.r41y b. 
i» «l«p»t. -u«>r« of «..t«y of .ny point of grM-nr. On. 
««Wr.. too. how -K* of th. -B-thom. .ffect- w.. invol«d^ 
in th. pot.tiw iiipro«-ent th. wirie. of t«.t. .how.. 

Sll*-'. .labor.t. pr.p.r.tlon of th. t..t for-.t rwr..!. 
« «iiH»S»... on th. P«t of «». «o-«i« Englirt t.«:h.r. to 
«.., f«» tr.dition.1 for-, of frting. B« coocla.ion. 
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clearly overstate tbm case and perhaps may even alBlead others 
into placing false trust in poorly constructed ■easures— con- 
elusion (fl) i» especially insidious. (See p. 10. above.) 

v.- The Experimental Itoissio ns Test at Oniversity of Clu j 

8ilva*8 confusion about the purposes and requir«s»nts of 
objective test^ i« cosswn aaong lUwian English teachers, but 
it is hard-»r to docurtent. Many assume that a question which asks 
a pupil to give a short ans*fer or to fill in a blank is i£SO facto 
\a good objective ite«. 

Xn 1»74. for instance, the Departnents of English and Teach- 
ing Methods at the Oniversity of Cluj experimented nith an "ob- 
jective test.-^ in it, ctudents were presented with an unidenti- 
fied 200-word passage fro* Daniel Defoe's novel itohi»*»on Crusoe 
and then asked 20 of the following sort of questions. Total test 
value was set at 12« points t 



Ite» 5. Give the essence of the 



((points) .upplJed) 



Ite» t. HiMit literary trend 

does this work belong to . 

(3 points) supplied) 



^nm results of the expei^W ^^^^^^^ ^^^i^f . 
focMtion about the nature of the test co«i8 from persona. - * 

Gtttioii with ite author, Bva iemlyen. . . . 



[a] 

Ztea 9* Piclt ont from the text Model: coine 

"~~ five words containing 

I ' the [a] Bound. Onder- 



Xine the latters by which ,c 

these sounds are repre- i5 lines 

sentedl ■ supplied) 

(5 points) 

Item 15."^ Translate! I I laid wait in this manner for 

— them. 



(2 points) 



Although these items are poor objective questions, they 
provide rather good examples of the oral recitation 'ftyle of 
testing coBB»nly used by classroom teachers to determine a pupil's 
trimester grade. In fact, the Cltfj questions are typical of the 
gerlpt followed In the traditional Romanian oral examination In 
BFL, both In classroom settings and In university admissions 

» 

competition. If for a variety of reasons unrelated to his actual 
co««and of the material (e.g.. Illness, distraction, fear) he does 
poorly In answering the oral questions, a pupil might receive a 
low or falling mark. (On the Romanian grading scale of 1 to 10. 
1 is' lowest, 10 hlghesti 5 Is a minimum passing mark on any test 
mt any level.) Or If the exam^r Is Inattentive (a special pro- 
blem in university admissions ixamlnatlons when the examining 
panel might question 50 Candida's In one day) or prefers a dif- 
ferent sytllstlc manner of translation, again the reliability 
of the test score given is questionable. 

m oral tasting teachers usually »*e -holistic" Judgments; 
in any event. It is rarely clear that the examiner Is focusing 
«A jmy partlcttlar items of tfte recitation since pupil responses 
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are often interspersed with teacher comments on a wide range 
of phonological, lexical, syntactic, biographical, literary, or 
critical issues which may suggest themselves during the oral 
testing. Most examiners with whom I have spoken say that their 
aim in oral testing is to examine the material of the lessons 
just studied (an amorphous goal) ; few have specific strategies 
for questioning. 

VI. criticism. Defense and Attempted Modifications of Some 
Traditional Forms of Testing 

Romanian texts on methodology of teaching that discuss 
objective tests as good classroom teaching and evaluation in- 
struments (Semlyen 1967 ; Semlyen and Filimon 1973; Galateanu . 
and Comiael 1975) do not have wide success in getting English 
teachers to replace or modify the traditional individual oral 
test (including the verbatim recitation of a memorixed portion 
from the textbook) , and to avoid impressionistic judgments 
about proficienty in English based on a complex of teacher con- 
cerns from tone of voice to posture. The trimester mark which 
each pupil receives continues to be the average of the scores 
he receives in such recitation opportunities and the grade he 
receives on a normally loosely defined written paper in which 
he may be asked to translate a passage, disciiss a literary or 
social theme that was covered in the textuook, or respond to 
four or five precise questions. (Munteanu 1971, pp* 138-139) 
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The nos-t serious practical constraint on the use of ob- 
jective tests in Romanian schools— a constraint which is not' 
written about—is the prohibition on duplicating any unofficial 
document in more than 10 to 15 copies. Old duplicating machines 
exist in some schools, but not for regular teacher use. In such 
circumstances, most teachers are deterred from creating tests 
that are useful only in multiple copies, since gaining approval 
from the proper agents (the Ministry of Propaganda) is an in- 
volved process, as I personally know. In* light of this fact, Silva 
■ight be less harshly criticised than others" for having pnly ten. 
Items on each of^her tests. If she did not apply for bureau- 
cratic approval to use a duplicating machine, she probably €yped 
each test, 6-8 copies at a time. 

In reviewing variable measures of student learning, Bejeiiaru 

(1975) singles out and criticizes the limitations of the Romanian 

# 

traditional oral testing methods in the following way: 
., • ■ 

(1) Pupils tend to recite verbatim from the textbooks. 

(2) Teachers tend to create ad hoc questions in a 
willy-nilly fashion. 

• (3) Tliere is time to qui* only 3 or 4 pupils per 
. . . '^Ass meeting while the other 30-35 remain 

) ^lamsivo on- lookers. • • 

'. (4|i'rSone pupils perform poorly under psycholegocal 
^»*^^«*ress when being cross-examined at the board 
•K^^f-Sa* front of their peers or alone in the un- 
"^lllMiiliar surroundings of the teachers' room. 

' NCSil * *'^teg^ee of subjectivity enters the evaluation 
4 ; '.' 'oX .oral tests. 



Moat practical teachers agre^ with Bejenaru's observations, but 
because of legal and physica). constraints they are not able to 
experiment with test styles that require individual copies of 
objective questions for e^ch' examinee. Consequently, experimen- 
tation with objective tests is for all practical purposes limited 
to researchers in pedagogy, kor whom duplication of materials, 

* ,. 

while not easy, is less difficult. 

Some testing researchers have chosen to experiment with 
developing better inter-reader reliability for traditional, essay- 
type tests, administered on«?e a term and involving about 5 questions 
for which pupils are given from 3 'to 4 minutes per question to 
wit;* precise fact-bearing answers from material they have studied, 
fiuch an experiment by Cerchez and Cerchez (1976), of course, 
avoided the constraint of having to have multiple copies of ob- 
jective tests. It r?id follow a carefully developed plan in which 
iver a four year period (1971-75) all teachers in an agricultural 
.liigh school, except for physical education staff, participated. 
«toe group of teachers was trained to read and grade pupil e8se.ys 
tcilven two times each year). by a di'scrcte-point scale developed 
Jointly by the researchers and the teachers. The, other group of 
taacher-readers evaluated the sa»e essays (which were anonymously 
coded) by the traditional -holistic- »ethod Romanian instructors 
Qse. Each essay was reviewed independently by. 2 or 3 readers 
£iom each group, and group composite grades were est^ulished by 
averaging the separate marjts. By correlating the two sets of 
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9rade8 on 2,162 written tests, Cerchez and Cerchez found that 
it is |K>s8lble to trace a pupil' s academic progress more care- 
fully by a less impressionistic, more systematic method»of 
evaluation, but they hasten to say that both forms of grading 
are necessary since some aspects of frogress are hard to quantify. 
Cerchez and Cerchez do not indicate if English was taught in the 
school where the experiment occurred. 

An experiment related to Cerchez and Cerchez 's is reported 
by Simlonescu (1975). In this case themes written on Romanian 
literary topics were graded by a discrete-p<!>int system, and 
altiiough Simionescu is sketchy about how many readers considered 
each essay (she only refers to "teacher" in trfe singular), sh^ 
reports that on the whole there is a stong correlation between 
the scores pupils said they thought they would earn and the marks 
the ussays were given. Like Cerchez and Cerchez, Simionescu 
argues that both precise and -holistic" grading should be done 
in Romanian schools since written work represents mastery of 
distinct skills (e.g., punctuation, organization, handwifiting) 
and the integration of these skills. The theme topics and the 
discrete-point scale Simionescu uses reveal her research design, 

For a freshman class at a high school for training elementary 
school teachers, Simionescu designated the topic "Sadoveanu: 
Famous Writer of Our Nation"; for the sophomore class in the same 
school the topic was -Folklore: Fountain of Inspiration in the 
works of Mihail Eminescu- ; and for the senior class the assigned 

* 
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subject was "George Calinescu: Historian and Literary Critic." 
All essay were to be written in Romanian. 

A grading scale was devised to agree with the usual 10- 
point one used in Romanian schools. The specific areas used 
in evaluation were: (a) handling of content— 0 to 5 points; 

(b) ability to synthesize and identify major issues— 0 to 1 point; 

(c) organization— 0 to 1 point; (d) style, orthography and physical 
appearance— 0 to 2 points; (e) personal writing style , imagination 
and unusual sensitivity— 0 to 1 point. As is obvious from the 
issues singled out for grading, a ^Qod deal of attention was 
given to matters of style and literary perception. These same 
concerns dominate the judgment of many EFL teachers who test 
through written assignments. 

' ^ There are no reported cases of any attempt , to standardize 
the marking of written papers in EFL classes. Howev6r, some 
scAse of the practice of essay marking can be gained from the 
writings by Levitchi (1971, 1972, 1973a, 1973b, 1975), who 
advocates extensive use of translation in teaching and testing 
EFL and yho exerts a profound influence upon the state of English 
studies in Rcanania, 

Teachers at every level of instruction use "the Levitchi 
style," consciously or unconsciously, by frequently employing 
grammar-translation methods in evaluating student performance even 
in what are presumed to be language classes. Experiments such 
as those of Dumitrescu and Galateanu, described above, and descrip- 
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tions of experiments in modif ied^audio-lingual methods such as 
given by Semlyen and Dragos (1971) are intended mote to counter- 
balance these pervasive grammar-tx^anslation methods than to de- 
velop serious research designs and experimental testing measures. 

■■ * ■ 

Nevertheless, complaints about testing are surfacing. A 
significant case of such a reevaluation of traditional testing 
procedures in EFL is reported in Bancila and Chitoran (1972) . 
In their description of the English uni>^ersity entrance examina- 
tion given at Bucharest in July 1971 to 685 candidates, these 
two members of the examining committee severely criticize the 
widespread use of translation and "literary appreciation" of 
texts in high schools to the detriment of students' learning how ... 
to use English in natural, communicative situations . Bancila 
and Chitoran, who is Professor of English and Dean of the Faculty 
of Germanic Languages at the University of Bucharest, conclude 
^ from the types of errors- candidates committed during tile examina- 
tion that methods o^ teaching and testing English in pre-university 
classes must be changed. 

Interestingly, along with their critique of the candidates' ^ 
preparation in English, Bancila and Chitoran give the most de- • • " 
tailed description available in the literature of the protocol and 
types of questions used in EFL testing in Romania: 

J 

f 

The written part of the examination consisted of 
two parts: 

1. the consideration of a chosen composition 
topic; 

2. the translation from Romanian into Englf r. 
of 12 sentences and statements . 
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The oral examination consisted of three parts: 

1. reading and translation of an excerpt 
from one of the selections contained in 
the textbooks for the llth and 12th 
grades identified in the Program of 
Studies for the Admissions Examinations 7 
lexical and cframmatical analysis of some 
words or underlined passages in the text. 

2. the transaltion from Romanian into English 
of two or three , short sentences. 

3. discussion of a literary topic connected 
with the authors and works mentioned in 
the Admissions Program. 

Although their discussion does not specify it, anyone 
familiar with Romanian university entrance examinations knows that 
ftft each student enters the examination room in turn, he draws his^ 
specific? questions from a pool of question tickets on the tdble 
before the examining panel, and then he has fifteen or twenty 
minutes to prepai^e his oral responses at the back of the room 
while the candidate before him is being examined. The pool pf^ 
questions contains items of varying difficulty, written by differ- 
ent members of the examining panel, on a wide range of topics 
based on the announced chapters of standard high school English 
textbooks. These announcements are publicized through the Entrance 
Examinations Catalogue published annually by the Ministry of Educa- 
tion (llinisterul Invatamintului, 1971, pp. -205-206 specify the 
material to be covered for English in the July 1971 sessions at all 
Romanian universities) . 

The Bancila-Chitoran report also offers as an appendix a^ 

*• * 

copy of the essay questions that were used and a full' copy of the 
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toa« sentences «hlch «re to.be rinaered iBto English: 

" Discuss one of the foUowing topics: 

. -siNbols of liberty in 19th centuiy English poetry. , 

«asure of man in a sociaUst society is 
^Ws Jespect for the -orking people. 

tte" t^l« ««nlan sentences for translation represent typical 

and iixical proble-s for lu-anian learners of English. 
IH* they do not for. a coherent paragraph. Further«,re. several ' 
the. are S«« fro- the s^^dard treasury of gra-ar-translat^on 
. ^tences that are often used as syntactic and se^ntic «zes in 
BFL classroom instructJ-on. 

U. a -ord. then, while the few objective tests experi«nted 
. ^th in R-anian En. settings contain serious flaws in construction 
«w. ass-ptionV, traditional essay exai-ination. also ccotinue to he 
^.tionable as valid, reliable -easures of student perforates .n 
English, except in isolated experiJKnts. 
VII. »»rlean E » r>ri.«t with an (ftjectl ve 

»..t of m in Konania 

intensifying national debate over the nature and adequacy ^ 
Of traditional evaluation p;ocedures «nlfe.ted Itself in a signxf- 
icant conference on university entrance ex^lnatlon. held at the 
.diversity of Wapoca on 7 Dece-ber 1974. Reporting on the 
^nts-^t the conference, Constantin (1975, note, that the pedagog- 
ical specialist m attendance discussed at length the adoption of 
Cjoctlv. standardlred tests to replace the usual oral examination 
^ Wrltt;n paper used to select fr- »on, the best candida.<,s those 
Who were to fill the limited classes in each discipline. 



After consideration of tlieoretical and practical issues re- 
gardissg objective tests (e^g.. Could such tests be .ore reliable 
than traditional examinations as «ea^es of overall proficiency 
and aptitude? Were entrance exaitinations of any sort redundant 
with the baccalaureate exaction at tbp end of the lyceu«? Would 
it be posa^wy to duplicate objective tests in sufficient nmbers?), 
the conferees generally agreed to urge "the judicious use of stand- 
ardized tests, but only when conditions *pere Most favorable." (p. 58) 
They also rec<«nded careful res^ch in the develop-^nt of tests 
which would enhance the selection of * candidates for university study. 

In January 1975, in the wake of this conference on testing, I 
arrived at the Oniversity of Cluj-Napoca with three of «y students " 
from Stockton State College, Pomona, New Jersey, having been invited 
by the Rector of the Oniversity to organize a joint ^rican-Romanian 
TEFL team which would develop h set of visual aids for use in teach- 
ing English in Romania. 

At our first audience with him, the Rector asked us to ^admin- 
ister an objective standardized 'test of EFL which had been produced 
in America to students who were majoring in English at the university 
Be had heard that such American- style tests were often used to de- 
termine if applicants to higher education in the United States- vere 
prepared fQf admission, and he wished to see if these tests would be 
helpful in the Romanian situation. He volunteered the full cooper- 
ation of the university. 
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As lie later discovered, nenbers of the University English 
OepartKnt were not as sanguine about the project since, it appear- 
ed,, they resented being told by the Rector without con.7ultation 
that foreigners— native English-spealcing ones at that— would be 
testing their students. Perhaps a portion of the indifference 
we met was caused by fear that students deeded well-prepared in 
English by the Department might do poorly on the test. 

Our eatplanat^n of the purpose of the test, its limitations, 
and the format of administration, delivered at a meeting of the 
English Department, received a cool welcome. Nevertheless, each 
faculty wanted a personal copy of the test. When we ex- 

plained that we had just enough copies for the testing and that 
customary security precautions prohibited a wide distribution of 

test copies, there was even less interest in the experiment among 

2 

some high ranking members of the Department. 

With the aid of the Dean of the Faculty of Philology, we 
pelected two groups to test, all freshman and junior English majors, 
and we also received appr^al tro3 the County Inspector of Schools 



one of our secret concerns was that it is not uncommon ^^r some 
RomaniJn instructors out of the best of motives to share ^^i^^ion 
«SJI?lS51s withfavorite students before evaluation sessions so they c 
dTJ^n 2e feaJIS th^t no reliable conclusions could be drawn from 
fSJ^T-ln/if secSitv were breached. What we lost xn cooperation fr< 
SS!^ gaineS^^Si ^tS^rs rLognizing, perhaps for the first time, i 
an objective test must be reliably administered. 
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— to test a group of 12th graders at a special English school in 
the city of Cluj. 

A. Th& Sobjects 

A group of 43 t**elfth grade pupils at Lyceum Nr. 1, "Ady- 
Sincai," was selected with about an equal number of subjects who 
were in the humanities (20 subjects) and science (23 subjects) 
tracks. JHost were girls; only 7 of the total were boys."' All 
had spent three and a half years at "Ady-Sincai, " where. a native 
English-speaking British teacher had been teaching conversation 
sections to all the pupils for a y^ar and a hzilf as part of the 
usual courses in English, sciences and art (taught in English) , 
imd French (the second foreign language for all the subjects in 
the experiment) • Lyceum Nr.l is a special English school (one of 
two in the country) in which aaany of the regular subject matter 
classes are taught in English. There are parallel French, Russian, 
and (German special schools. Admission to all special schools is 
highly competitive. Twenty-five of the 43 subjects had studied 
English in school for 8 years, 7 for 5 years, 6 for 4 years, 3 for 
7 years and 1 each for 6 years and 10 years. The maternal tongue of 
37 of the pupils was Romanian, 5 Hungarian, and 1 Saxon German. 

The group of 91 university freshman English majors included 



3 . 
In Romania most language specialists are women since men are ex- 
pected to enter a trade or science. Because it is a special English 
school, •Ady-Sincai's*' enrollment is not typical of that of rabst Romania 
high schools, but is representative of the female/male distribution of 
groups which major in English. 
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who ca«e fro« varixms high schools throughoat Booaxxia, hot 
were from the region of Transylvania. Forty-six had French 
as their academic minor, 16 Romanian, 11 German, 7 Hungarian, 6 
Bnssian, and 5 did irit identify their minor language. One subject 
had studied EngUsh for 14 years, 5 for. 10 years, 10 for 9 years, 
29 for a years, 2 for 7 years, 6 for 6 years, 14 for 5 years,' 17 for 
4 years, 3 for 3 years, 2 for only 2 years, and 2 did not indicate 
the time they had studied English formally. All biit 12 of the 
freslmien subjects were female. 

Of the university junior group of 67 subjects, 50 were first- , 
language Romanian speakers, 13 had Hungarian as their maternal lang- 
uage, and 4 indicated that Saxon German was their first language. 
The academic minor language for 29 of the juniors was French, for 
16 German, for 13 Romanian, for 5 Russian, and for 4 Hungarian. 
Among the subjects, 25 had studied English in school for 7 years, 
12 for II years, 8 for 6 years, 7 for 8 years, 5 for 10 years, 4 
for 9 years, 3 for 5 years, and 1 each for 14, 12, and 3 years. 
There were 10 male students in the experimental group. 

B. The Instruments 

Form B (Revised 1965) of the Michigan Test of English Lang - 
uage Proficiency (MTELP) was selected mainly for the following 
reasons: (1) It is a standardized objective test with a respectable 
reliability coefficient (Spearman-Brown: .965 and Kuder-Ri chardt r 



•894), Standard Error of Measurenent (S-B: 3.35 and K-Rs 3.54), 
and aean raw scores %Aen administered to t«o nonaing groups (M = 
75.35, on a group of 150 applicants to Aaerican universities, and 
M » 81.44, on a gro<^> of 100 recooMended applicants to Aaierican 

* 

universities). (2) It has reasonable content and construct validity, 
as described in the Manual accoii?)aniring the test,. and contains 
objectively scored items on grammar (40 multiple-choice, short 
context questions) , vocabulary (40 multiple-choice substitution and 
ael^ition or £ill-in-the-blank item§l , and reading (four 200-word 
paragraphs followed by five multiple-choice items that can be answered 
correctly only if the subject has understood the paragraphs.) (3) The 
test Manual provides for the grouping of subjects by equated scores 
on predictive norms established by the University of Michigan for 
admitting foreign students into restricted or unrestricted curricular 
sequences. (4) The test fit the available time in the Romanian 
schedule of classes, and sufficient test material? were available 
for the size of the groups to be tested. 

Because it- seemed desirable to test the subjects' listening 
comprehension and because the MTELP does not conte^in an aural com- 
prehension subtest, the first part (20 items) of the 1972 edition 
of the Michigan Enc^lish Placement Test (MPT) was selected. Further- 
more, since the MPT is a non-diagnostic, objectively scored test 
designed for quick placement of non-native English speaking students 
into homogeneous ability groups, it was decided to keep the scores 



earned on the MPT separate' from those earned on the MTELP, and 
to use the MPT scores as very rough indicators of the aural com- 
prehension ability of the total test population. The fact that 
-he MPT containes only 20 items on listening and has not been normed 
added weight to thi^ decision. The longer and normed Michigan Test 
»f Anral Comprehension . Form A, «as considered for use but rejected 
since the time alloted for test sessions was only two hours. The 
Ustening items on the MPT are of two •types: questions that re- 
quire seleciiion of an appropriate reply from three choices and 
statements for which a suitable paraphrase has to be selected from 
three choices. 

C. The Test Administration a nd Scoring 

Each subject was give a personal copy of the sample questions 
and answer sheet that accompa^iies the MTELP in order to familiarize 
herself with the format of the test. With the approval of the Chair- 
woman of the English Department and senior professors, instructors 
who were greatly interested in the experiment went over the sample 
question and answer sheet with seminar-size groups of subjects (about 
20 persons) two days before the admini.8tration of the tests. In 
the high school similar familiarization sessions were held by the 
English teachers. All those who conducted these sessions had them- 
selves had an earlier familiarization session with the American 
members of the experiment. 

Tte three test groups were administered the tests under as nearly 
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siiailar conditions as possible on three separate dates: Univer- 
sity freshmen were tested on 31 January 1975, University juniors 
on 6 February 1975, and high school saniors on 10 February 1975. 
During the testing special care was taken to prevent cheating (a 
customary habit among some students that Romanian teachers tend to 
overlook) by having the two American Fulbright lecturers in the 
English Department assist in the proctoring along wit^ the few 
Bomanian faculty: members who came. Also, a tests and measurements 
Romanian specialist from the University's psychology Department 
introduced the test procedure in Romanian to ensure that everyone 
understood what %/as to be done. ^ 

The MPT was administered first accordi'ng €o the precise directions 
in the test Manual. Some ranking members of the English Depart- 
ment had objected to my plan to give the 
grounds that the students' ears were not 

pronunciation of American English, but I decided the objection was 
not substantial since university English Uudents at Clu j had had 
exposure to native-speaking American Fulbright lecturers since 1970. 
in contrast, the teachers at the high scho6l were eager for me to ^ 
speak the cues< since they felt that their pupils should be able. to'., 
understand a native, having taken conversation classes with the^. 
British t-g^cher and having listened to American voices on audio tapes 
in classes. . 

The scoring of the answer sheets was done immediately after 
each of the three ^Ministrations . A team of graders composed << 
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oral cues myself on the 
accustomed to my native 



senior psychology majors who had stndied testing theory was 
hand-picked for dependability and concern for professional con- 
fidence by the university psychologist who helped administer the 
tests. I personally supervised the scoring. 

Preliminary analysis of thie test rtrsults was done on a Rtwian- 
ian coaiputer program (FELIX C-256) , but because more elaborate 
use of con5»uter facilities in Romania was unauthorized (computer 
center.^ are carefully guarded) , these preliminary calculations 
fiere check in the United States on programs in SPSS (Sie 1975); 
statistics calculated by hand have >een based on Payne (1968). 

V- 

D'. Discussion of the Results 

The Spearman-Brown split half reliability coefficient was 
ccanputed on the obtained data for the 100-item.MTELP and the results 
are as follows: r = .813, with a SE^^^^ of 3.59; raw total mean = 
75.53. with a SD of 9.93 and N of 201. 

For the 20- item MPT Listening Subtest, r = ,546, SExneas = 1.68, 
the raw total mean = 15.67, with a SD of 2.5/ and N of 2Q1. 

The low r for the MPT Subtest was expected and suggests that 
no great confidence can be placed in the Subtest results. While 
a higher r for the MTELP would be desirable, we can still use the 
test results with a relatively high degree of confidence since 
"most reliability coefficients over .70 are probably within an 
acceptable range." (Payne 1968, p. 136) 



. . ^Uje_l (at Ijack) displays statistics on test results by 
cla£s level and by subtest, raw total, and equated total scores. 
Equated scores represent raw total scores trajasformed to be equiv- 
alent to 'marks earned on Form A of the MESLP and on Form A of the 
itichiqan Test of Aural Comprehension by a norming group of 284 
foreign sttidents at the University of Kansas and the University of 
Midiigan in 1961. 

Tests riin on the between-level differences on s^ibtest scores 
made by the Eomanian test popluation reveal no significant dif- 
ferences except between mean reading scores earned by the 12th Grad 
.ars and the nniversitv jianiorsN Table 1 , Column D) . Significance 
level = .05 (Buchanan 1974, pp. 95-97) . As a result, for all prac- 
tical purposes the data yielded by the three test groups can be 
considered together since the Standard Error of Measurement of the 
test instrument can account for variance between observed and true 
scores. '^Ejneas = 3-35. Thus, the part scores and standard devia- 
tions for the entire test population, might be respresented as in 
>. 

the following Table: 

Summary of Means and Standard Deviati ons for All Groups 

» = 201 



Section of the MTELP and MPT 


Items 


Mean Score 


SD 


KTELP Grammar Svibtest 


40 


33.78 


3.84 


HJELP Vocabulary Subtest 


40 


29.25 


4.19 


MTJFJLP Reading Subtest 


20 


12.55 ' 


3.93 


MTELP Raw Total 


I'.OO 


75.53 


9.93 


MTELP Equated Total ^ 


100 


80.14 


/8.63 


MPT Listening Subtest 


20 


15.67 / 


2.57 



The SD noted in each cell of liable 1 suggests the spread 
of the scores away from the mean for each grade level. A 
'smaller SD irepresents a tighter clustering of scores about the 
niean. One might expect the scores of isrrc advanced students 
to gather nearer the mean, as weaker students either improve 
oi^:irbpbut, and, in fact, observation shows this to be so in 
most cases. The fact that the SD f or ^University Freshmen some- 
times is larger than the equivalent SD for 12th Graders in high 
school (hence a broader spread of scores among the University 
group) , might be understood as a result of a more heterogeneous 
grouping of University Freshmen, most of whom had not, studied 
in special English schools like *Ady-Sincai." 

Table 3 displays Pearson correlation coefficients for 
part and total scores on the tests. Among the subtest scores 
of the MTELP the strongest correlation occurs, as one might ^ 
expect, between vocabulary and reading scores: r = .5897. 
overall, the correlations among part-scores of the MTELP are ^ 
all substantial or marked ^nd indicate that each part is function 
ing to measure different English-language abilities in the test 
population. The strongest relationship between any single part- 
score on the MTELP and the MPT Listening Comprehension Subtest - 
involves the Grammar section of the MTELP: r = .3129. This 
coefficient suggests a slight relationship between what the two 
subtests are measuring, yet as we said above, because of the 
low reliability of the 20-item Listening Subtest this correlation 
cannot be trusted . - 



Finally, Table 4 interprets the HTELP equated total scores 
by class using the English lang^^age proficiency norms d^^veloped 
at the university of Michigan over «.ny years.. (MPT Lis ning " 
scores do not have a bearing here.) Because the KPEI^ is not 
a test-of students' achievement in a particular course of studies 
bat a general' estimate of English language proficiency, the ' _ - 
equated total scores can be used as guides to a student's ability 
to pi^sue academic work at an American university. The proficiency 
norms developed at Michigan, where large numbers of non^native 
speakers of English have studied, serve the function of distin- 
guishing the level of English proficiency a student needs in 
order to have a reasonable chance of success in different academic 
programs in an American "university . The recommendations by 
degree programs in Table 4 are based, on the relationship between 
academic performance by the norming group during'the first 
semester of study and their scores on the Michigan battery. Table 
4 Shows the number and percentage of the Romanian subjects by 
Ilass Who scored in each recommenced range for the three American 
curricular levels: Group I : Undergraduates in liberal arts and 
education. Group 11= Graduates and undergraduates in engineering, 
mathematics and scientific fields that depend heavily 6n laboratory 
work. Group III: Graduate humanities and social sciences . (English 
and American literature, law, political science, philosophy, etc.) . 
^s one might anticipate, a higher percentage of university juniors 
^uld be considered for acceptance by most 'schools at the University 
of Michigan than of university freshmen, and a higher percentage of 
university freshmen than of high school 12th graders. 
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E. Interpretations and Conclusions 

One of the most interesting facts, siir facing from the 
data provided by the tests taken by the . 201 Romanians is that 
there is no significant difference in the quality of performance 
on the tests by university English majors and by high school 
l-2th graders who have studied for four years in a special English 
school. 

At first glance this fact might suggest that little ox o 
progress in English proficiency occurs during university^ training. 
However, when one remembers that the pupils in "Ady-Sincai" 
were receiving special language 'treatment— a curriculum in which 
English is the language of instruction for^many subject-matter 
courses as well as in the conversation course taught by a native 
speaker of English--and when one 'further recalls that^most 
university English majors had not attended such a special English 
school prior to. matriculation, it seems that the university 
students were able to reach through their training the lev^ of 
proficiency achieved in the special English school, or conversely, 
that the experiences of English training in a special school like 
"Ady-Sincai" is of university caliber. 

If the MTELP were administered to another group of English 
pupils who were comparable to the "Ady-Sincai" pupils in every ^ 
respect except that they were not in a special English school 
(and/or to an entering class of university English majors) , and 
if a parallel form of the MTELP were administered to the oria^jjal 
91 university Freshman subjects before they graduate in June 1978 f 
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this hypothesis could be checked. 

Finally, in their pursuit of more reliable measures in 
testing, Romanian researchers might focus on the degree of . . 
proficiency in English of pupils in special schools as opposed 
io ^t of pupils in schools following more traditional pro- 
grams. If repeated objective testing reveals o^e group to be 
significantly better prepared than the other, the adoption of 
standarized evaluation measures could have an ipiportant effect 
not only upon the viniversity entrance examination but also 
upon day-to-day classroom practice. \ 
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